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ABSTRACT 

Methods are described for visualizing and 
characterizing user interactions with hypermedia systems; they have 
been found to be effective in several developmental and evaluation 
studies. Analyzing the patterns of user interaction makes it possible 
to evaluate the design and usefulness of such systems. Path algebras 
can be used to describe and compare routes users take through 
hypermedia systems. Directed graphs are another method of 
constructing a network representation of user paths using the 
Pathfinder algorithm. Another approach is to construct an image based 
on interaction data with the Toolbook program. Fifty-seven data files 
from a previous study were entered into the Toolbook program to 
generate a map of each user's interaction. Six gifted middle-school 
students, six teachers, and six doctoral students in instructional 
technology were asked to categorize the images and to associate them 
with course grades of the users. Although sub j ect s could not 
generally associate images with grades, they were able to place them 
in identifiable categories* which suggests the potential of the 
method for depicting user paths. Nine figures illustrate the 
approaches. (Contains 16 references.) (SLD) 
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The unprecedented freedom for users to control the scope and sequence of their 
interactions with hypermedia systems presents many challenges to those who design and 
study these systems in educational settings. Early efforts to develop hypermedia systems 
revealed that the inherent node-link structure is both advantageous and problematic (see 
Conklin, 1987). But when users have the freedom to follow any of a multitude of link 
permutations, disorientation often results. Further, without appropriate training, novice 
users do not possess the strategies necessary for effective "browsing" of large hypermedia 
documents (Duffy & Knuth, 1991). Many designers, therefore, advocate that features such 
as visual maps, search facilities, and guided tours be included in hypermedia systems to 
alleviate some of these problems (e.g. Hammond, 1989; Laurel, 1990, 1991). 

/ 

With the emergence of hypermedia systems as a major architecture for educational 
and other information-oriented software comes the related problem of how to document and 
analyze user interactions with these systems for the purposes of research and evaluation. 
Many hypermedia systems to date have employed a "browsing" interface, but alternative 
approaches are also emerging (See Nelson & Palumbo, 1992). Regardless of the type of 
interface, many questions can be generated when studying the interactions of users with 
hypermedia systems. For example, how many users chose to follow a particular link, and 
why was one link chosen over another? How does the choice of one link affect choices of 
subsequent links? When are graphic images, animations, and video segments accessed? 
What user tasks are appropriate for guiding interaction with the system? Are there 
patterns of navigation that lead to more complete learning? These and many other 
questions need answers when designing, developing, and evaluating hypermedia 
applications for education and other settings. 

A wealth of user interaction data can be easily collected within many hypermedia 
development environments in order to study aspects of the interface, including the nature of 
user navigation patterns, the time spent at each node, and the use of help and other 
orienting facilities. The data can represent the paths a user follows through the system, 
and the choices made at each node in the system. The problem is that because of the 
nature of this data, traditional methods of analysis such as surveys or pretest-posttest 
designs, are not particularly effective for determining usability or comparing alternative 
interface designs. Researchers have had to develop new techniques for analyzing patterns of 
user interaction in order to evaluate the design and effectiveness of hypermedia systems (e. 
g. Misanchuk & Schwier, 1992). There is a need to categorize and compare groups of users 
in order to compare the effectiveness of alternative system features, as well as describing 
characteristics of interaction by individual users within the same system. This paper will 
discuss several methods for visualizing and characterizing user interactions with 
hypermedia systems that have been found effective in several development and evaluation 
studies. 

Characterizing user interactions using path algebras 

Characterizing the interactions of individual users, or comparing groups of users, can 
be accomplished using several methods derived from mathematical set and graph theories 
(Backhouse & Carre, 1975; Carre, 1971). In the first method, path algebras are employed 
to describe and compare the routes users take through hypermedia systems (Alty, 1984). 
Users may foliuw a variety of types of paths through the two-dimensional space of a 
hypermedia database (Canter, Rivers & Storrs, 1985). Using simple programming 
techniques, it is possible to "trap" and save in data files the sequence of nodes visited by the 
user, along with the more specific user actions taken at individual nodes, such as clicks on 
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buttons, menu selections, or viewing graphic images, animations, or audio and video 
segments. 

The data files can be analyzed using algorithms that extract the nature of the 
"subpaths" of the total path followed by the user. The types of subpaths users follow 
through hypermedia systems can include rings, loops, paths, and spikes (See Figure 1). As 
described by Canter, Rivers, and Storrs (1985), paths are movements between nodes in a 
linear sequence from one node to another. Spikes occur when a user moves through a 
number of nodes, and then returns along the path to the initial node. Loops and rings are 
similar in that the user moves through a set of nodes that eventually returns to the original 
node in a circular pattern. Rings are created when a user loops through several patterns 
that visit nodes that were previously accessed. The path that an individual user follows 
while interacting with a hypermedia system gives an indication of the nature of that 
interaction at different points in time. Of course, the types of paths a user might follow will 
be determined in part by the organizational structure of the system. Hypermedia systems 
organized hierarchically will tend to generate paths with many spikes, as the user moves 
down a "branch" of the tree and then back to the "root". A linear organization for the 
hypermedia system will tend to produce many paths. Rings or loops tend to indicate that a 
user is revisiting nodes, perhaps to search for information, or perhaps because of 
disorientation. 

Figure 1. Types of user paths through hypermedia systems. 



Loop 




The frequencies of each type of path followed during an individual users interaction 
provides a basis for comparison between individuals and groups. These frequencies can be 
recorded for each user, and common parametric statistical procedures can be employed to 
compare the mean number of each path type for groups of users in order to determine 
whether the frequencies were significantly different. This approach provides a means for 
analyzing individual users' interactions that can be more precise than merely determining 
the frequency that one node is accessed from another node (Misanchuk & Schwier, 1992). 
There is also the possibility of comparing groups of users that is not afforded by other 
methods that focus on individual user access patterns. In this way, alternative interface 
design strategies can be compared using quantitative methods that may reveal several 
advantages of one design over another. 

/ 
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This method was employed with data collected in a prior study (Nelson, 1991) to 
further characterize the interaction patterns of various users who possessed different levels 
of prior knowledge for the domain presented in a hypermedia system. Two groups of users 
with high and low prior knowledge of the content completed one of three tasks with a 
hypermedia document (browse, search, or study). The results indicated that prior 
knowledge influenced interaction patterns (e.g., there were more "spikes" for individuals 
with lower prior knowledge). In addition, the results of this analysis also confirmed a design 
assumption that was made, namely, that definitions and examples of unfamiliar terms 
needed to be provided to users with low' prior knowledge, but that these definitions and 
examples would not be accessed by users with high prior knowledge. The large number of 
"spikes" followed by some users was the result of their access of definition and example 
nodes for some concepts in the document. It appears that this method of analysis of user 
interactions with hypermedia systems can be useful in testing design alternatives. 
Extensions of this technique might utilize similar algorithms embedded within the 
hypermedia system to detect in real time some of the various interaction patterns, and to 
provide system interventions or guidance based on such analyses. 

Characterizing user interactions using directed graphs 

The second method employs graph theory to construct a network representation of the 
paths taken by individual users through a hypermedia document, and to compare the 
networks as generated by the Pathfinder algorithm (Schvaneveldt et al., 1985). Pathfinder 
was originally developed to study the associative characteristics of human memory, and has 
been applied in a variety of research studies (Schvaneveldt, 1990). The algorithm is 
applied to proximity data that represents participants' ratings of the degree of relatedness 
between a set of concepts. The resulting proximity matrices are analyzed by Pathfinder to 
construct a network representation. Correlations can then be computed between 
individuals' networks to determine the degree to which the networks are similar. 

When used to study navigation patterns in hypermedia, this method assumes that the 
more frequently a link is traversed by a user, the more direct is the association in the 
resulting Pathfinder network that represents the interaction pattern. To analyze the data, 
proximity matrices are constructed to represent the relative "proximity" of a node to every 
other node in the hypermedia document. In this case, proximity is defined as the number of 
times a node was visited through a link from another node. As shown in Figure 2, the raw 
data is used to construct the proximity matrix by analyzing the user's sequential moves 
between nodes. Each move from one node to another results in an increase in the frequency 
value of the corresponding cell in the matrix. So a move from node 1 to node 7 would 
increase the matrix cell at row 1-column 7 by one, and a move from node 7 to node 3 would 
increase the matrix cell at row 7-column 3 by one, and so on through the sequence of the raw 
data. After individual matrices have been developed for each user, group data can be 
derived by calculating the average value for each cell in the matrix (See Figure 2), 




Figure 2. Deriving a proximity matrix from raw data, and averaging across a group of users. 
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Once th^ proximity matrices* are established, the Pathfinder algorithm can extract the 
representative network from the average matrix, as shown in Figure 3. A computer-based 
tool for analyzing data using the Pathfinder algorithm is available (Interlink, 1990). The 
relatedness coefficients used to plot various Pathfinder solutions from different data sets can 
be correlated to determine the degree of similarity of the paths taken by individuals or 
groups. The correlation values indicate the degree to which different groups of users followed 
similar paths through the hypermedia document. This method was also applied to the data 
from the study cited above. Correlations between the groups suggested that characteristics 
of the task may have influenced the interaction patterns of users. At least for this data set, 
searching and browsing tasks produced patterns that were more similar than for the study 
task, as indicated in the comparatively high correlations. This method may also be 
extended by hypermedia authors to specify the initial organization of the nodes and links in 
a hypermedia document (McDonald, Paap & McDonald, 1990). 

A Picture of the Users* Interaction 

Another approach to visualizing the interaction path of an individual user is to construct 
an image based on the interaction data. The interaction that we wanted to represent was the 
amount of time spent on each screen, the direction of links, and the number of times that a 
screen was visited. In order to do this, each screen of information that was visited by the user is 
represented as a rectangle. The more time that is spent on a node, the larger the rectangle 
becomes. When a learner links from one piece of information to another, a line segment is 
dr^wn with an endpoint on the originating node and an arrowhead on the destination node. 
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Further, the more times a learner visits a node, the darker it becomes with a range cf 4 (white, 
light gray, dark gray, black). The resulting image would then represent the interaction of that 
user. 



Nelson's (1991) data were used in the Toolbook program to generate an image of the 
interaction each student had with the system. There were a total of 57 images produced (one 
for each subject). In the Nelson study, each subject was a student in an undergraduate 
educational psychology class. The system that they examined was a hypermedia program on 
the field of educational psychology. There were three groups. The Study group was told that 
they will have a test on the content in the program. The Browse group was told to evaluate the 
program because it might be used in later classes. The Index group was asked to find 
definitions to a list of words. 

Figure 3. Pathfinder solution for hypothetical data. 
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Method 

The 57 data fih'S for the subjects in the Nelson (1991) study were entered into the 
Toolbook program to generate a map of each individual learner's interaction with the 
program. These 57 images were printed and copies were made. The images were given to 
three different groups: Gifted middle school students (n=6), teachers in an introductory 
educational computing course (n=6), and instructional technology doctoral students (n=6). 
The first two groups were given the 57 images and some paper clips and were instructed to 
place the images on a table and try to categorize the images into 3 to 5 groups. They were 
then asked to write a description of the procedure they followed to create the categories and 
to describe the categories. The third group did the same thing, except they were told what 
the images represented and each image indicated which group from the Nelson study the 
image was from. 

The only performance data available from the Nelson study was the course grade 
each subject received. While this is a very indirect measure of the impact of the instructional 
package, it was the only one available. Grades in the sample were either A, B or C. 
Unfortunately, for a variety of reasons, only 38 of the 57 images could be associated with a 
grade. The categories generated by the groups in the current study were then used in a chi- 
squared analysis. One dimension was the grade received by the learner and the other 
dimension was the category generated by the subjects in this study. A total of 18 chi- 
squared tests were conducted. 
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Results 

Results indicated that there was only one subject whose categorization strategy 
resulted in a statistically significant result in regards to the grade received in the course(see, 
Figure 4). The categories were: (W) "Has more dark squares than any other kind" (see, 
Figure 5); (X) "Has more white squares than any other kind" (see, Figure 6); (Y) "Has more 
polka dot Qight gray) squares than any other kind;" and, (Z) "Two highest patterns were 
tied." Unfortunately, expected values are very low in the lower categories and would result 
in questionable conclusions. However, given the exploratory nature of this research 
methodology, it is worth taking a closer look. 



Figure 4. Summary table and observed frequencies for the only significant categorization 
scheme based on grade in the course and created by N2. 
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It appears that th.* first category (W) has many people who received an A in the 
class. This first category can be described as a pattern where the user returned to most 
screens at least 3 times. A!so, of interest is that the majority of the people who got a B in 
the course fall into category X. This category can be described as a pattern where the 
majority of the screens in the program were visited only once. This result points toward an 
interesting question for future research. That is, does the pattern where the learner returns 
to the same screen many times result in better learning? And, if this statement is true, can 
a system be designed that adjusts the level of user control based 'on this interaction pattern. 
There were not enough people who received a C or fell into the last two categories to conclude 
anything about them. 
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Figure 5. An example of category W created by N2. 




Figure 7. Summary table and observed frequencies for the only categorization scheme that 
was significant for grade as based on treatment group. 
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Figure 6. An example of category X created by N2. 




A related question to the one described above is how do the categories generated by 
subject N2 relate to the experimental treatments in the original Nelson (1991) study? 
Figure 7 depicts the results of this analysis. While the result is not significant, a pattern 
similar to grade can be seen. That is, there are a large number of subjects that are both in 
the study treatment and fall into the pattern where the user returned to most screens at 
least 3 times (category W). Likewise, the pattern where the majority of the screens in the 
program were visited only once (category X) is well represented by the Browse and Index 
treatments. While it is premature to make any kind of definitive statement about whether 
these patterns are at all related to learning, it does hold some promise for a more rigorous 
experimental examination. 

Chi-squared analyses were performed on the categories generated by the subjects in 
this study with the treatment groups in the Nelson study. The results indicated that all of 
the Instructional Technology doctoral students* categories were significantly related to the 
treatment groups. This is not surprising given that the doctoral students were aware of the 
groups and from which group each image came. There wero also two subjects that were not 
informed of what the images represented that resulted in significant categories. These 
categories were generated by the teacher group in the current study and the results are 
depicted in Figures 8 and 9. 
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Figure 8. Summary table and observed frequencies for significant categorization scheme 
based on treatment group. 
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Figure 9. Summary table and observed frequencies for significant categorization scheme 
based on treatment group. 
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For subject N4, categories T and V seem to define people in the treatment groups 
best. That is, category T does not include anyone from the study treatment group. 
Similarly, category V accounts for the majority of the study group (10 out of 17). The 
categorization scheme used by subject N4 is based on the shading of nodes (just as subject 
N2). In order to be included in category T, the picture should contain "no more than 2 
different shades." Category V for subject N4 was that there were "no more than 4 different 
shades of boxes." Upon examining the nature of the images for the category T, we noticed 
jit most all of these images had the 2 shades of light gray and white (all but 2 of them). 
This is not an interaction pattern that would indicate that these learners were returning to 
screens to re-examine information. On the other hand, the V category shows a pattern of 
behavior that certain screens are visited once, others twice, etc. While it is pure conjecture 
on our part, this would indicate that the users in this category were judging the value of the 
information and returning to screens that were important and not returning to screens that 
were not. 

Subject N6 used a categorization scheme that was based on both frequency of 
returning to a screen (shade) and the number of screens visited (the number of nodes on the 
image). Categories O and Q can be collapsed into a single category where there were "no 
dark gray or black boxes and 6 or fewer boxes." Category R would be the opposite (the 
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image contained dark gray or black boxes and it had more than 6 boxes). It is interesting to 
note that frequency of visiting screens is such an important attribute (it is an aspect of each 
of the categories described). Future research ought to examine this navigation pattern more 
closely. 

Conclusions 

The three methods for analyzing user interactions with hypermedia systems 
described above have been successfully employed to compare the patterns of navigation 
exhibited by groups of users in several studies. The last analysis seems to also provide 
some potential for further research more directly related to navigation patterns as they 
relate to learning. The analysis of peoples interactions with hypermedia systems is quite 
complex. The capability to visualize this interaction allows a researcher the ability to 
examine many different attributes simultaneously within the image. The results indicate 
that there appears to be merit to this type of analysis. 

At this time, the methods in this paper appear promising for further research into 
usability studies of hypermedia systems, as well as more basic research into various 
theories of human-computer interaction. The latter research will also require additional 
methodologies in order to obtain the rich descriptions of interaction with hypermedia that is 
necessary to further develop theories and principles for the design of such systems. 

References 

Alty> J. L. (1984). The application of path algebras to interactive dialogue design. Behavior 

and Information, Technology, 3 (2), 119-132. 
Backhouse, R. C, & Carre, B. A. (1975). Regular algebra applied to path-finding problems. 

Journal of the Institute of Mathematics and Its Applications, 15, 161-186. 
Canter, D., Rivers, E . & Storrs, G. (1985). Characterizing user navigation through complex 

data structures. Behavior and Information Technology, 4 (2), 93-102. 
Carre, B. A. (1971). An algebra for network routing problems. Journal of the Institute of 

Mathematics and Its Applications, 7, 273-294. 
Conklin, J. (1987). Hypertext: An introduction and survey. IEEE Computer, 20 (9), 17-41. 
Duffy, T. M., & Knuth, R. A. (1930). Hypermedia and instruction: Where is the match? In 

D. Jonassen and H. Mandl (Eds.), Designing hypermedia for learning. Heidelberg, 

FRG: Springer-Verlag. 

Hammond, N. (1989). Hypermedia and learning: Who guides whom? In H. Maurer (Ed.), 

Computer Assisted Learning. New York: Springer-Verlag. 
Interlink. (1990). KNOT-Mac: Knowledge network organizing tool for the Mac [Computer 

Program]. Las Cruces, NM: Interlink, Inc. 
Laurel, B. (1990). Interface agents: Metaphors with character. In B. Laurel (Ed.), The art 

of human-computer interface design (pp. 355-365). Reading, MA: Addison- Wesley. 
Laurel, B. (1991). Computers as theater. Reading, MA: Addison-Wesley. 
McDonald, J. E., Paap, K. R., & McDonald, D. R. (1990). Hypertext perspectives: Using 

Pathfinder to build hypertext systems. In R. W. Schvaneveldt (Ed.), Pathfinder 

associative networks: Studies in knowledge organization, (pp. 197-212). Norwood, NJ: 

Ablex. 

Misanchuk, E. R., & Schwier, R. A. (1992). Representing interactive multimedia and 

hypermedia audit trails. Journal of Educational Multimedia and Hypermedia, 1 (3), 
355-372. 



ERLC 



639 f 



Nelson, W. A. (1991, April). Effects of prior knowledge and task characteristics on 

information access in hypertext documents. Paper presented at the annual meeting of 

the American Educational Research Association. Chicago, IL. 
Nelson, W. A., & Palumbo, D. B. (1992). Learning, instruction and hypermedia. Journal of 

Educational Multimedia and Hypermedia, 1 (3). 287-300. 
Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge 

organization. Norwood, NJ: Ablex. 
Schvaneveldt, R. W., & Durso, F. T., Goldsmith, T. E., Br-en, T. J., Cooke, N. M., Tucker, R. 
G., & DeMaio, J. C. (1981). Measuring the structure of expertise. International 
Journal of Man-Machine Studies, 23, 699-728. 



•3 



640 




